Image processing apparatus

ABSTRACT

An image processing apparatus includes detection circuitry and processing circuitry. The detection circuitry, in operation, detects a position on a sensor that is pointed to by a pointer. The processing circuitry, in operation, generates three-dimensional data that includes the position detected by the detection circuitry, positions a two-dimensional surface relative to the position detected by the detection circuitry, and applies rendering to, at least, part of the three-dimensional data that is arranged on one side of the two-dimensional surface to be displayed on a display screen.

BACKGROUND 1. Technical Field

The present disclosure relates to an image processing apparatus thatenables a three-dimensional (3D) (three-dimensional) image to be shownon a display screen from coordinate data that represents detectionoutput of a position pointed to by a pointer such as finger orelectronic pen.

2. Description of the Related Art

Technologies are available that allow for generation of 3Dthree-dimensional images and showing them on a display screen. Forexample, Japanese Translations of PCT for Patent No. 2002-513480discloses a technology that converts a two-dimensional (2D) photographicpicture into a 3D modeled object and also converts the 3D modeled objectback into the original 2D photographic picture.

If the user performs pointing input with an electronic pen to draw aline, for example, it is common that the drawn line appears on a displayscreen as an image shown on the display screen using coordinate datathat represents detection output of a position pointed to by a pointersuch as finger or electronic pen. A tablet is known that associates theelectronic pen, for example, with a paint brush. Also in this case, ifthe user makes a pointing input with an electronic pen to draw a line, aline that matches a tip width of the associated brush appears on thedisplay screen.

Incidentally, when one draws with a brush and paints on a canvas, thepaints on the canvas have thicknesses. Digital inks have no thicknesseswhen one draws a stroke on a sensor of a position detection section witha pointer such as an electronic pen or finger. Although it is possibleto achieve representation that artificially makes the inks look as ifthey had thicknesses by applying a shadow, it is difficult to representtextures.

BRIEF SUMMARY

In light of the foregoing, it is desirable to provide an imageprocessing apparatus that enables translation of pointing input by apointer, through comparatively simple process, so that thicknesses canbe represented as if the stroke was drawn, for example, with an oilpaint.

According to an embodiment of the present disclosure, there is providedan image processing apparatus that includes detection circuitry andprocessing circuitry. The detection circuitry, in operation, detects aposition on a sensor pointed to by a pointer. The processing circuitry,in operation, generates three-dimensional data that includes theposition detected by the detection circuitry, positions atwo-dimensional surface relative to the position detected by thedetection circuitry, and applies rendering to, at least, part of thethree-dimensional data that is arranged on one side of thetwo-dimensional surface to be displayed on a display screen.

In the image processing apparatus according to the above embodiment, theprocessing circuitry generates three-dimensional data that includes theposition detected by the detection circuitry. For example, theprocessing circuitry uses spheres as a three-dimensional shape, andgenerates spherical three-dimensional data that includes the positiondetected by the detection circuitry at the center of the sphere.

Then, the processing circuitry positions a two-dimensional surfacerelative to the position detected by the detection circuitry. Then, theprocessing circuitry applies rendering to, at least, part of thethree-dimensional data that is arranged on one side of thetwo-dimensional surface to be displayed on the display screen.

The image appearing on the display screen using this image informationis something similar to 3D three-dimensional graphics appearing on atwo-dimensional screen. This makes it possible for the image processingapparatus according to the above embodiment to generate, through simpleprocesses, image information that enables translation of pointing inputby a pointer so that it appears puffy as if it was drawn, for example,with an oil paint.

The present disclosure provides an advantageous effect in that imageinformation is generated that enables translation of pointing input bythe pointer detected by the detection circuitry so that a thickness canbe represented as if it was drawn, for example, with an oil paint.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an overall outline of an embodiment ofan image processing apparatus according to the present disclosure;

FIG. 2 is a diagram for describing a configuration example of a positiondetection section used in an embodiment of the image processingapparatus according to the present disclosure;

FIGS. 3A to 3C are diagrams for describing a processing example of theposition detection section used in an embodiment of the image processingapparatus according to the present disclosure;

FIG. 4 is a diagram illustrating a hardware configuration example of amain body of the image processing apparatus in an embodiment of theimage processing apparatus according to the present disclosure;

FIGS. 5A to 5D are diagrams for describing main parts of an embodimentof the image processing apparatus according to the present disclosure;

FIG. 6 is a diagram illustrating a flowchart for describing an exampleof processing operations of the main body of the image processingapparatus in an embodiment of the image processing apparatus accordingto the present disclosure; and

FIGS. 7A and 7B are diagrams for describing a modification example ofthe embodiment of the image processing apparatus according to thepresent disclosure.

DETAILED DESCRIPTION

An embodiment of an image processing apparatus according to the presentdisclosure will be described below with reference to the accompanyingdrawings.

FIG. 1 is a diagram illustrating a configuration example of the imageprocessing apparatus according to the present embodiment. In thisexample, the image processing apparatus includes a main body 100 of theimage processing apparatus and a tablet terminal 200 having a displayfunction. The main body 100 of the image processing apparatus includes,for example, a computer. Then, in this example, the tablet terminal 200is connected to the main body 100 of the image processing apparatus viaa cable 300.

In this example, the tablet terminal 200 includes a display section 201that includes, for example, a liquid crystal display (LCD) panel.Further, the tablet terminal 200 includes a capacitive positiondetection section 202 on the back of the display section 201 in thisexample. As an example of a pointer, an electronic pen 203 is suppliedwith the tablet terminal 200.

The electronic pen 203 in this example includes a signal transmissioncircuit to send a position detection signal to the position detectionsection 202. Further, in this example, the electronic pen 203 includes awriting pressure detection section to detect a writing pressure appliedto a tip thereof. Writing pressure information detected by the writingpressure detection section is sent to the position detection section 202of the tablet terminal 200.

In this example, the position detection section 202 includes a positiondetection sensor that is approximately the same size as a display screen201D of the display section 201. The position detection section 202detects the position on the position detection sensor pointed to by theelectronic pen 203 as two-dimensional XY coordinates by detecting theposition detection signal. Further, the position detection section 202of the tablet terminal 200 receives writing pressure information fromthe electronic pen 203, thus detecting the writing pressure from thereceived information. Then, the tablet terminal 200 pairs each piece ofcoordinate data of the position pointed to by the electronic pen 203detected by the position detection section 202 and the writing pressureapplied to that pointed position and sends this information pair to themain body 100 of the image processing apparatus.

A control item menu is displayed on the display screen 201D of thetablet terminal 200 in addition to an image display field 201Px. Thecontrol item menu, shown in a frame area around the image display field201Px, allows for the user to select and set parameters for generatingan image. The main body 100 of the image processing apparatus generatesinformation to be shown in the control item menu in the frame areaaround the image display field 201Px, sending the information to thetablet terminal 200 and showing it on the display screen 201D.

The user selects desired parameters and options from the control itemmenu in the frame area around the image display field 201Px by pointingwith the electronic pen 203. The main body 100 of the image processingapparatus recognizes where the parameters and options of the controlitem menu in the frame area around the image display field 201Px areshown. Therefore, the main body 100 of the image processing apparatusdetects which parameters and options have been selected by the user fromcoordinate information of the position pointed to by the electronic pen203.

In the example illustrated in FIG. 1, a 2D/3D switching button 201SWappears in the upper frame area of the image display field 201Px. The2D/3D switching button 201SW enables selection of whether to draw a 2Dor 3D image in the image display field 201Px with the electronic pen203. The example illustrated in the frame area of FIG. 1 shows a case inwhich the 2D/3D switching button 201SW is in the 3D position. In thisexample, a color palette 201CP with a plurality of display colors isdisplayed in the upper frame area of the image display field 201Px. Byselecting an arbitrary display color from among a plurality of colors,the user can select a display color for the image at the coordinateposition pointed to by the electronic pen 203.

Then, in this example, a brush type selection section 201BP appears onthe left of the image display field 201Px to select a brush type to beassociated with the pointer. By selecting an arbitrary brush or pen typefrom among a plurality thereof, the user can draw an image with theselected brush or pen type at the coordinate position pointed to by theelectronic pen 203.

Further, in this example, a light source change section 201LE, abrightness adjustment section 201BRT, a color adjustment section 201CC,and a 3D reference shape selection section 201PS appear on the right ofthe image display field 201Px. The user can change, with the lightsource change section 201LE, at least either the light source positionor type during rendering for generating a 3D image. As used herein,rendering or “applying rendering to” means generating an image to bedisplayed on a display screen, for example, the display screen 201D andit may include surface rendering and volume rendering. A renderingsection 110 can make an adjustment so that rendering is performed inreal-time in accordance with at least either the light source positionor type changed by the light source change section 201LE. Further, thelight source change section 201LE may automatically change at leasteither the light source position or type during rendering for generatinga 3D image. For example, light source positions and types suitable fordifferent seasons or times may be set in advance so that at least eitherthe light source position or type changes automatically. Also in thiscase, the rendering section 110 can make adjustment so that rendering isperformed in real-time in accordance with at least either the lightsource position or type changed by the light source change section201LE.

Further, the user can adjust, with the brightness adjustment section201BRT, the brightness of the 3D image appearing on the display screen.Still further, the user can adjust, with the color adjustment section201CC, the chromaticity of the 3D image appearing on the display screen.

The 3D reference shape selection section 201PS is used to select a 3Dreference shape during generation of 3D volume data from coordinateinput of the position pointed to by the electronic pen 203. In theexample illustrated in FIG. 1, the 3D reference shape selection section201PS has different selectable shapes. For example, sphere, cone,truncated cone, and other shapes are shown, and the user can select oneof these shapes. It should be noted that the 3D reference shapes thatcan be selected with the 3D reference shape selection section 201PS arenot limited thereto. Instead, selectable shapes may include not only acylinder but also columnar bodies with tetragonal, pentagonal, hexagonaland other cross-sections. The 3D reference shapes may further includecolumnar bodies with cross-sections of arbitrary shapes.

The main body 100 of the image processing apparatus acquires coordinatedata and writing pressure information of the position pointed to by theelectronic pen 203 from the position detection section 202 of the tabletterminal 200. The main body 100 of the image processing apparatusdetects the selected parameters and control items from the acquiredcoordinate data and writing pressure information, generating displayimage data in accordance with the detected parameters and control itemsand sending the data to the tablet terminal 200. The tablet terminal 200shows the image information received from the main body 100 of the imageprocessing apparatus in the image display field 201Px in the displayscreen of the display section 201.

Configuration Example of Position Detection Section of Tablet Terminal200

The position detection section 202 according to the present embodimentincludes a sensor 204 and a pen pointing detection circuit 205 connectedto the sensor 204 as illustrated in FIG. 2. The sensor 204 has a sensorsurface (pointing input face) that is comparable in size to the displayscreen 201D of the tablet terminal 200. The sensor 204 is lighttransmissive and includes first and second conductor groups 211 and 212.

The first conductor group 211 includes a plurality of first conductors211Y₁ to 211Y_(m) (where m is an integer equal to or greater than 1)that extend, for example, horizontally (x-axis direction) and arearranged parallel to each other in the y-axis direction with a givenspacing therebetween. On the other hand, the second conductor group 212includes a plurality of second conductors 211X₁ to 211X_(n) (where n isan integer equal to or greater than 1) that extend in a directionintersecting the direction of extension of the first conductors 211Y₁ to211Y_(m). In this example, the second conductors 211X₁ to 211X_(n)extend vertically (y-axis direction) in a direction orthogonal to thedirection of extension of the first conductors 211Y₁ to 211Y_(m) and arearranged parallel to each other in the x-axis direction with a givenspacing therebetween.

In the description given below, the first conductors 211Y₁ to 211Y_(m)and the second conductors 212X₁ to 212X_(n) will be referred to as thefirst and second conductors 211Y and 212X, respectively, if it is notnecessary to distinguish between the individual conductors.

The pen pointing detection circuit 205 includes a selection circuit 221,an amplification circuit 222, a band-pass filter 223, a detectioncircuit 224, a sample-hold circuit 225, an analog to digital (AD)conversion circuit 226, and a control circuit 220. The selection circuit221 serves as an input/output interface with the sensor 204.

The selection circuit 221 selects a conductor from among each of thefirst and second conductor groups 211 and 212 based on the controlsignal supplied from the control circuit 220. The conductors selected bythe selection circuit 221 are connected to the amplification circuit222. The signal from the electronic pen 203 is detected by the selectedconductors and amplified by the amplification circuit 222. The output ofthe amplification circuit 222 is supplied to the band-pass filter 223,thus extracting only the frequency component of the signal sent from theelectronic pen 203.

The output signal of the band-pass filter 223 is detected by thedetection circuit 224. The output signal of the detection circuit 224 issupplied to the sample-hold circuit 225 that samples the signal at agiven timing using a sampling signal from the control circuit 220 andstores the sampled value, after which the sampled value is convertedinto a corresponding digital value by the AD conversion circuit 226.Digital data supplied from the AD conversion circuit 226 is read by thecontrol circuit 220 for necessary processing.

Thanks to the program stored in an internal read-only memory (ROM), thecontrol circuit 220 operates in such a manner as to send a controlsignal to each of the sample-hold circuit 225, the AD conversion circuit226, and the selection circuit 221. Further, the control circuit 220calculates the coordinates of the position on the sensor 204 pointed toby the electronic pen 203 from the digital data supplied from the ADconversion circuit 226, outputting that position coordinate data toother image processors in the tablet terminal 200.

FIGS. 3A to 3C are timing diagrams for describing a given pattern signalreceived by the sensor 204 of the position detection section 202 fromthe electronic pen 203 according to the present embodiment. Theelectronic pen 203 according to the present embodiment includes a signaltransmission circuit 203S and a control circuit 203CTL. Thanks to acontrol signal supplied from the control circuit 203CTL, the signaltransmission circuit 203S repeatedly outputs a signal having a givenpattern.

FIG. 3A illustrates an example of a control signal supplied from thecontrol circuit 203CTL that controls the signal transmission circuit203S of the electronic pen 203. For a set time period in which thesignal remains high, the electronic pen 203 successively outputs atransmission signal (alternating current (AC) signal at a givenfrequency) in the form of a burst signal from the signal transmissioncircuit 203S (continuous transmission period in FIG. 3C) as illustratedin FIG. 3B.

The continuous transmission period is long enough for the pen pointingdetection circuit 205 of the position detection section 202 to detectthe position on the sensor 204 pointed to by the electronic pen 203. Forexample, the continuous transmission period is long enough to scan allthe first and second conductors 211Y and 212X once or more andpreferably a plurality of times or more.

During the continuous transmission period, the control circuit 203CTL ofthe electronic pen 203 detects the writing pressure applied to the pentip, thus finding the writing pressure as a multiple bit value (binarycode) from the detection result. Although not illustrated, the writingpressure detection section having a known configuration described inJapanese Patent Laid-Open No. 1993-275283 or Japanese Patent Laid-OpenNo. 2011-186803, for example, may be applied for use as the writingpressure detection section. Further, for example, a semiconductorelement with variable capacitance in accordance with the writingpressure may also be applied as disclosed in Japanese Patent Laid-OpenNo. 2013-161307.

Then, when the continuous transmission period ends, the control circuit203CTL of the electronic pen 203 inserts a start signal and begins atransmission data period as illustrated in FIG. 3C. During thetransmission data period, the control circuit 203CTL controls thecontrol signal (refer to FIG. 3A) to a high or a low level at givenintervals (Td), thus amplitude shift keying (ASK) modulating thetransmission signal from the signal transmission circuit 203S. Thetransmission signal from the signal transmission circuit 203S may bemodulated into an on off keying (OOK) signal rather than using ASKmodulation.

At this time, the control signal is typically at the high level in thefirst given interval (Td) following the continuous transmission periodand used as a start signal as illustrated in FIG. 3C. This start signalis a timing signal to allow for accurate determination on the subsequentdata transmission timings by the position detection section 202. Itshould be noted that a burst signal during the continuous transmissionperiod may be used as a timing signal rather than this start signal.

The electronic pen 203 sends writing pressure data, transmission data(digital data) made up of a given number of bits, one after anotherfollowing the start signal during the transmission data period. In thiscase, when the transmission data (binary code) is ‘0’, the electronicpen 203 pulls the control signal to the low level and does not send anytransmission signal. On the other hand, when the transmission data(binary code) is ‘1’, the electronic pen 203 pulls the control signal tothe high level and sends a transmission signal. The electronic pen 203repeatedly sends a pattern signal made up of a continuous transmissionperiod and a transmission data period as described above at intervalsbased on control performed by the control circuit 203CTL.

In the pen pointing detection circuit 205 of the position detectionsection 202, for example, the control circuit 220 supplies a selectionsignal to the selection circuit 221. The selection signal is used toselect the second conductors 212X₁ to 212X_(n) one after another. Wheneach of the second conductors 212X₁ to 212X_(n) is selected, the controlcircuit 220 reads the data output from the AD conversion circuit 226 asa signal level. Then, if the signal level of none of the secondconductors 212X₁ to 212X_(n) reaches a given value, the control circuit220 determines that the electronic pen 203 is not on the sensor 204,repeatedly selecting the second conductors 212X₁ to 212X_(n) one afteranother.

If signal levels equal to or higher than the given value are detectedfrom the second conductors 212X₁ to 212X_(n), the control circuit 220stores the number of the second conductor 212X with the highest detectedsignal level and those of the plurality of second conductors 212X aroundthe second conductor 212X with the highest signal level. Then, thecontrol circuit 220 selects the first conductors 211Y₁ to 211Y_(m) oneafter another and reads the signal levels supplied from the ADconversion circuit 226 by controlling the selection circuit 221. At thistime, the control circuit 220 stores the number of the first conductor211Y with the highest detected signal level and those of the pluralityof first conductors 211Y around the first conductor 211Y with thehighest signal level.

Then, the control circuit 220 detects the position on the sensor 204pointed to by the electronic pen 203 from the numbers of the second andfirst conductors 212X and 211Y with the highest detected signal levelsand those of the pluralities of second and first conductors 212X and211Y therearound.

When the signal level detection is over following the selection of thefinal first conductor 211Y_(m) by the selection circuit 221, the controlcircuit 220 waits for the end of the continuous transmission period ofthe signal transmitted from the electronic pen 203. When the controlcircuit 220 detects a start signal following the end of the continuoustransmission period, the control circuit 220 performs an operation toread transmission data such as writing pressure data, thus receivingsuch transmission data as an ASK or OOK signal. Then, the controlcircuit 220 pairs at least coordinate data of the position pointed to bythe electronic pen 203 and writing information and sends thisinformation pair to the main body 100 of the image processing apparatus.

Configuration Example of Main Body 100 of Image Processing Apparatus

FIG. 4 is a block diagram illustrating a configuration example of themain body 100 of the image processing apparatus. As described earlier,the main body 100 of the image processing apparatus includes a computerin this example. A control section 101 includes a central processingsection (CPU). A position detection section interface (denoted as I/F inFIG. 4) 103, a display controller 104, a coordinate and other dataanalysis section 105, a control pointing detection holding section 106,a 2D image data generation section 107, a volume data generation section108, a display image information generation section 109, and a renderingsection 110 are individually connected to the control section 101 via asystem bus 102.

The position detection section 202 of the tablet terminal 200 isconnected to the position detection section interface 103. Whencoordinate data and writing pressure information are received from theposition detection section 202, the position detection section interface103 sends the received coordinate data and writing pressure informationto the coordinate and other data analysis section 105.

Further, the display section 201 of the tablet terminal 200 is connectedto the display controller 104. Display image information generated bythe display image information generation section 109 is supplied to thedisplay section 201 of the tablet terminal 200 via this displaycontroller 104 and appears on the display screen 201D thereof as will bedescribed later.

The coordinate and other data analysis section 105 detects whether thecoordinates of the received coordinate data fall within the imagedisplay field 201Px of the display section 201 of the tablet terminal200 or within the surrounding frame area thereof. Then, the coordinateand other data analysis section 105 sends the coordinates in the framearea around the image display field 201Px to the control pointingdetection holding section 106.

The control pointing detection holding section 106 determines, from thecoordinate data received from the coordinate and other data analysissection 105, which control items, i.e., the 2D/3D switching button201SW, the color palette 201CP, the brush type selection section 201BP,the light source change section 201LE, the brightness adjustment section201BRT, the color adjustment section 201CC, and the 3D reference shapeselection section 201PS, was selected by pointing. The control pointingdetection holding section 106 stores the determined results as controlpointing data.

In the meantime, when the coordinates of the coordinate data fall withinthe image display field 201Px, the coordinate and other data analysissection 105 refers to the status of the 2D/3D switching button 201SWstored in the control pointing detection holding section 106. When the2D/3D switching button 201SW is in the 2D position, the coordinate andother data analysis section 105 supplies the coordinate data to the 2Dimage data generation section 107. On the other hand, when the 2D/3Dswitching button 201SW is in the 3D position, the coordinate and otherdata analysis section 105 supplies the coordinate data to the volumedata generation section 108. It should be noted that, in this case, thecoordinate and other data analysis section 105 sends coordinate data andwriting pressure data as a pair as described earlier.

The 2D image data generation section 107 generates image information ofa line drawing that matches a stroke, successive positions pointed to bythe electronic pen 203. In this case, the line thickness changes inaccordance with writing pressure information. Further, when a wide brushis selected in the brush type selection section 201BP, the 2D image datageneration section 107 generates image information made up of lineswhose width matches the brush width. The 2D image data generationsection 107 supplies generated image information to the display imageinformation generation section 109.

The display image information generation section 109 converts thereceived image information into display image information, informationto be shown on the display screen of the display section 201, supplyingthe converted display image information to the display section 201 viathe display controller 104. Therefore, when the 2D/3D switching button201SW is in the 2D position, a line drawing image that matches thestroke, successive positions pointed to by the electronic pen 203,appears on the display screen of the display section 201.

When the 2D/3D switching button 201SW is in the 3D position, the volumedata generation section 108 receives coordinate data and writingpressure information from the coordinate and other data analysis section105, thus generating volume data. In this case, the volume datageneration section 108 generates volume data within thethree-dimensional shape selected in the 3D reference shape selectionsection 201PS in the present embodiment. In that case, the volume datageneration section 108 associates a given position within the selectedthree-dimensional shape with the coordinate data position supplied fromthe coordinate and other data analysis section 105 and generates volumedata as a three-dimensional shape of the size according to the writingpressure of the writing pressure information.

Then, in the present embodiment, the brush type selected in the brushtype selection section 201BP is identified. The three-dimensional shapeaccording to the writing pressure of the writing pressure information isvaried in accordance with the identified brush type. That is, forexample, volume data may be generated so that even if the writingpressure is the same, the three-dimensional shape is larger in size whendrawn with a thick brush than a thin brush.

If the three-dimensional shape selected in the 3D reference shapeselection section 201PS is, for example, a sphere, the volume datageneration section 108 places coordinates (Xi, Yi) acquired from thecoordinate and other data analysis section 105 at a given position inthe sphere such as at a center (center of gravity) Os thereof asillustrated in FIG. 5D, thus generating spherical volume data of aradius r according to the writing pressure of the writing pressureinformation. In this case, the magnitude of the radius r variesdepending on the selected brush type as described earlier. It should benoted that it is not necessary for each brush type to have a differentradius r and that the two or more brush types may have the same radiusr.

It should be noted that when position pointing input is made to theposition detection section 202 with the tip of the electronic pen 203 incontact with the position pointing input face on the sensor 204 of theposition detection section 202 (surface of the display screen 201D inthis example), it is only necessary to set, for example, Z=0 as the zcoordinate in the z-axis direction orthogonal to the x- and y-axisdirections. Then, writing pressures that match the respective coordinatepositions are applied to the electronic pen 203.

It should be noted that even if the three-dimensional shape selected inthe 3D reference shape selection section 201PS is other than a sphere,volume data can be similarly generated by placing the coordinates (Xi,Yi) acquired from the coordinate and other data analysis section 105 atthe center (center of gravity) of each of the three-dimensional shapesand by generating volume data that matches the writing pressure of thewriting pressure information and whose size matches the selected brushtype.

It should be noted that the present disclosure is not limited to placingthe coordinates (Xi, Yi) acquired from the coordinate and other dataanalysis section 105 at the center (center of gravity) of each of thethree-dimensional shapes. Instead, it may associate the coordinates (Xi,Yi) acquired from the coordinate and other data analysis section 105with a given position set within the three-dimensional shape.

Volume data generated by the volume data generation section 108 is sentto the rendering section 110. The rendering section 110 sets atwo-dimensional surface (e.g., a two-dimensional plane or atwo-dimensional curved surface), applying rendering to the volume datathat is on one side of the set two-dimensional surface of all the volumedata sent based on the coordinates acquired from the coordinate andother data analysis section 105. As a two-dimensional surface to be set,a virtual surface can be used which matches the display screen 201D onthe sensor surface (sensor plane) of the display section 201. That is,the volume data that is on one side of the set two-dimensional surfacecorresponds to the portion bulging up on the display screen 201D.

That is, in this example, the rendering section 110 uses a targetrendering data generation section 111 to set a circular plane PLo thatis parallel with the two-dimensional surface (X-Y plane in thisexample), the pointing input face of the sensor 204, and whose Z=0 firstwithin the received volume data as illustrated in FIG. 5D based oncoordinates (Xi, Yi, Z=0) acquired from the coordinate and other dataanalysis section 105. Then, the target rendering data generation section111 considers one side of the set plane PLo, and in this example, theportion of the volume data that is on the front side (shaded in FIG. 5D)to be the one subjected to rendering. It should be noted that we assumein this example that a light source is provided on the front side of theplane PLo during rendering. Therefore, the volume data to be subjectedto rendering is the portion that is on the front side of the plane PLothat is exposed to more light thanks to the light source.

Next, a rendering execution section 112 of the rendering section 110applies surface rendering, in this example, to the target portion ofvolume data generated by the target rendering data generation section111. Then, the rendering section 110 sends the rendered portion ofvolume data to the display image information generation section 109. Inthis surface rendering, the light source position specified in the lightsource change section 201LE is referred to. At the same time, necessaryadjustments are made using the adjustment levels selected in thebrightness adjustment section 201BRT and color adjustment section 201CCto obtain an image that represents a thick-looking digital ink texture.

The description given above focused on volume data generation andrendering for a single point (single set of coordinates) whose positionwas pointed to by the electronic pen 203. However, when the actualposition pointing input made by the user with the electronic pen 203 isa stroke, successive positions pointed to by the electronic pen 203 maybe used.

For example, a description will be given below of a case in which theuser moves the tip of the electronic pen 203 to draw a straight line inthe x-axis direction with the tip of the electronic pen 203 in contactwith the position pointing input face on the sensor 204 of the positiondetection section 202.

In this case, we assume that the writing pressure at each of thecoordinate positions during movement of the tip of the electronic pen203 in the x-axis direction changes as illustrated in FIG. 5A. Then,each set of coordinates and each piece of writing pressure informationare supplied to the volume data generation section 108 via thecoordinate and other data analysis section 105 during movement of thetip of the electronic pen 203.

The volume data generation section 108 places each set of coordinatessuccessively sent from the coordinate and other data analysis section105 at the center (center of gravity) Os of a sphere in this example asillustrated in FIG. 5D, generating spherical volume data of the radius raccording to the writing pressure of the writing pressure information.In this case, volume data generated for the respective sets ofcoordinates partially overlaps each other as illustrated in FIG. 5B. Asfor these overlaps, the portion of volume data generated later ispreserved. As a result, the volume data generation section 108 generatesvolume data that matches the three-dimensional shape as illustrated inFIG. 5C in response to stroke input from the electronic pen 203,supplying the volume data to the rendering section 110.

Then, the target rendering data generation section 111 of the renderingsection 110 sets a plane PLc within the volume data as illustrated inFIG. 5C based on each set of coordinates acquired from the coordinateand other data analysis section 105. The plane PLc includes all the setsof coordinates, is parallel with the two-dimensional surface (X-Y planein this example), the pointing input face of the sensor 204, and hasZ=0. Then, the target rendering data generation section 111 considersone side of the set plane PLc, and in this example, the portion of thevolume data that is on the front side (shaded in FIG. 5C) to be volumedata subjected to rendering.

Next, the rendering execution section 112 of the rendering section 110applies surface rendering, in this example, to the target portion ofvolume data generated by the target rendering data generation section111. Then, the rendering section 110 sends the rendered portion ofvolume data to the display image information generation section 109.

The display image information generation section 109 converts thereceived image information into display image information, informationto be shown on the display screen of the display section 201, supplyingthe converted display image information to the display section 201 viathe display controller 104. Therefore, when the 2D/3D switching button201SW is in the 3D position, an image is shown on the display screen ofthe display section 201 that looks three-dimensionally puffy inaccordance with the stroke, successive positions pointed to by theelectronic pen 203.

It should be noted that the components making up the main body 100 ofthe image processing apparatus in FIG. 4 as described earlier, namely,the display controller 104, the coordinate and other data analysissection 105, the control pointing detection holding section 106, the 2Dimage data generation section 107, the volume data generation section108, the display image information generation section 109, and therendering section 110, may be implemented in the form of circuitryincluding a combination of hardware and software, for example, softwarefunctional sections that can be executed by the program stored in thestorage apparatus (not shown), and any of the sections can be combinedtogether or further divided into sub-sections according to eachimplementation.

Example of Processing Operation Flow of Main Body 100 of ImageProcessing Apparatus

A description will be given below of an example of a processingoperation flow of the main body 100 of the image processing apparatuswith reference to the flowchart shown in FIG. 6.

It should be noted that we assume in the description given below thatthe process in each step of the flowchart in FIG. 6 is performed by thecontrol section 101 by executing each of the display controller 104, thecoordinate and other data analysis section 105, the control pointingdetection holding section 106, the 2D image data generation section 107,the volume data generation section 108, the display image informationgeneration section 109, and the rendering section 110 as circuitingincluding a combination of hardware and software, for example, asoftware functional section using the program stored in the storagedevice (not shown).

The control section 101 determines whether data has been received fromthe position detection section 202 (step S101). If determined otherwise,other processes are performed (step S102), and then the process isreturned to step S101.

If the control section 101 determines in step S101 that data has beenreceived from the position detection section 202, the control section101 analyzes the received coordinate data (step S103), determiningwhether the coordinate data is pointing input made to draw an image inthe image display field 201Px or that made in the control item menu inthe frame area (step S104).

If the control section 101 determines in step S104 that the coordinatedata is pointing input made in the control item menu in the frame area,the control section 101 determines which of the following control items,namely, the 2D/3D switching button 201SW, the color palette 201CP, thebrush type selection section 201BP, the light source change section201LE, the brightness adjustment section 201BRT, the color adjustmentsection 201CC, or the 3D reference shape selection section 201PS, wasselected by pointing, storing the control item found as a result of thedetermination (step S105). Then, the control section 101 returns theprocess to step S101 following step S105, repeating the processes fromS101 onward.

Further, if the control section 101 determines in step S104 that thecoordinate data is pointing input made to draw an image in the imagedisplay field 201Px, the control section 101 refers to the status of the2D/3D switching button 201SW, determining whether it is a 2D or 3D imagethat can be drawn (step S106).

If the control section 101 determines in step S106 that it is a 2D imagethat can be drawn, the control section 101 performs necessary processesto draw a 2D image in the same manner as described above (step S107) andthen returns the process to step S101, repeating the processes from S101onward.

On the other hand, if the control section 101 determines in step S106that it is a 3D image that can be drawn, the control section 101generates volume data based on the coordinate data as mentioned earlier(step S108). In this step S108, the brush type selected in the brushtype selection section 201BP and the 3D reference shape selected in the3D reference shape selection section 201PS are recognized prior togeneration of volume data. Then, volume data for the recognized 3Dreference shape is generated at the size according to the writingpressure and the recognized brush type.

Following this step S108, the control section 101 sets thetwo-dimensional surface PLo within the three-dimensional shape formedwith the volume data as described earlier, generating the portion ofvolume data located on one side of the set surface PLo as data to besubjected to rendering (step S109).

Next, the control section 101 recognizes the light source positionspecified in the light source change section 201LE and also recognizesthe adjustment levels selected in the brightness adjustment section201BRT and the color adjustment section 201CC, applying surfacerendering, in this example, to the target volume data generated in stepS109 using these recognition results (step S110). Then, the controlsection 101 sends the 3D image that has undergone rendering to thedisplay section 201, showing the image on the display screen of thedisplay section 201 (step S111).

Effect of Embodiment

As has been described up to this point, if one simply generates volumedata in a given 3D reference shape based on coordinate data, the imageprocessing apparatus according to the present embodiment enables showingof pointing input in the form of a stroke made with the electronic pen203 such that it has thickness as if it was drawn with an oil paint.Moreover, in the above embodiment, the size of the 3D reference shapeserving as a reference for generating volume data changes in accordancewith the writing pressure applied to the electronic pen 203. As aresult, the user can change the oil paint thickness in accordance withthe writing pressure, thus enabling representation of a painting inaccordance with the actual brush stroke of the user.

Then, in the above embodiment, the size of the 3D reference shapeserving as a reference for generating volume data can be changed inaccordance with not only the writing pressure applied to the electronicpen 203 but also the brush type selected by the user, also allowing theuser to represent painting in a desired manner in this respect.

Then, in the above embodiment, the 3D reference shape can be selected bythe user. As a result, the above embodiment also has an advantageouseffect in that the user can also select the oil paint thickness.

Other Embodiment or Modification Example

It should be noted that the processes performed by the target renderingdata generation section 111 of the rendering section 110 are not limitedto those described above. For example, the target rendering datageneration section 111 sets a two-dimensional surface based oncoordinate data of positions pointed to by the electronic pen 203 anddeforms volume data generated by the volume data generation section suchthat the volume data can be arranged on the two-dimensional surface.That is, the target rendering data generation section 111 generatesvolume data to be subjected to rendering typically by cutting the volumedata with the two-dimensional surface that includes the positionspointed to by the electronic pen 203 and arranging the cut surface onthe two-dimensional surface.

Further, in the above embodiment, as a two-dimensional surface based oncoordinate data of positions pointed to by the electronic pen 203, afixed two-dimensional surface that has this coordinate data, is parallelwith the sensor's x- and y-axis directions and has Z=0 is used. However,the set position of the two-dimensional surface based on coordinate dataof positions pointed to by the electronic pen 203 may be user-variablerather than fixed. For example, the set position of the two-dimensionalsurface may be changed in the direction normal to the surface (e.g.,negative z-axis direction) as illustrated in FIG. 7A. In the exampleillustrated in FIG. 7A, the two-dimensional surface (alternate long andshort dashed line 301) shown in FIG. 5D is moved down by AZ in thedirection normal to the surface (e.g., negative z-axis direction) toobtain a two-dimensional surface (solid line 302). In this case, thereis volume data under a two-dimensional plane that has not been subjectedto rendering. Therefore, rendering is applied again to the volume dataon the upper side of the two-dimensional surface including the volumedata under the two-dimensional plane.

Alternatively, the set position of the two-dimensional surface may bechanged by tilting the surface by a given tilt angle θ as illustrated inFIG. 7B. In the example illustrated in FIG. 7B, the two-dimensionalsurface (alternate long and short dashed line 301) shown in FIG. 5D isrotated by the angle θ around a straight line running along the y-axisdirection that includes coordinate data (Xi, Yi) of positions pointed toby the electronic pen 203 as a rotational center to obtain atwo-dimensional surface (solid line 303). It should be noted that if thetwo-dimensional surface is tilted, a change in the direction normal tothe surface (z-axis direction) is permitted. Further, it is a matter ofcourse that the two-dimensional surface may be tilted in any direction.Also in this case, there is volume data under a two-dimensional planethat has not been subjected to rendering. Therefore, rendering isapplied again to the volume data on the upper side of thetwo-dimensional surface including the volume data under thetwo-dimensional plane. In either case, the rendering section 110 canmake adjustment so that rendering is applied in real-time to thetwo-dimensional surface whose set position has been changed.

Further, a 3D and 2D mixed image may be drawn on the display screen ofthe display section 201 with the electronic pen 203 by switching the2D/3D switching button 201SW.

Still further, in the above embodiment, the rendering execution section112 of the rendering section 110 applies surface rendering to the targetvolume data. Instead, however, the rendering execution section 112 mayapply volume rendering.

Still further, although, in the above embodiment, the brush type and the3D reference shape are selectable independently from each other, adifferent 3D reference shape may be associated in advance with eachbrush type.

Other Modification Example

Although, in the above embodiment, an electronic pen is used as apointer, a finger may be used instead. Further, although a capacitivepen is used, an electromagnetic or other kind of pen may also be used asan electronic pen.

Although the image processing apparatus includes the main body 100 ofthe image processing apparatus and the tablet terminal 200, theconfiguration thereof is not limited thereto. Instead, the main body 100of the image processing apparatus may be integral with the tabletterminal 200. That is, the image processing apparatus according to thepresent disclosure may be configured in any way as long as it includesthe functions of the main body 100 of the image processing apparatus andthe functions of the position detection section 202 of the tabletterminal 200.

It is to be noted that the present disclosure is not limited to theforegoing embodiments, and that various changes can be made withoutdeparting from the spirit of the present disclosure.

What is claimed is:
 1. An image processing apparatus comprising:detection circuitry, which, in operation, detects a position on a sensorthat is pointed to by a pointer; and processing circuitry, which, inoperation, generates three-dimensional data that includes the positiondetected by the detection circuitry, positions a two-dimensional surfacerelative to a display screen on which the three-dimensional data isdisplayable, and applies rendering to, at least, rendering target dataof the three-dimensional data that is arranged on a first side of thetwo-dimensional surface to be displayed on the display screen, therendering target data being defined based on the two-dimensionalsurface, the rendering not being applied to the three-dimensional datathat is arranged on a second side of the two-dimensional surface, thesecond side of the two-dimensional surface being opposite the first sideof the two-dimensional surface.
 2. The image processing apparatus ofclaim 1, wherein the processing circuitry, in operation, positions thetwo-dimensional surface within the three-dimensional data based on theposition detected by the detection circuitry and applies rendering to,at least, the rendering target data of the three-dimensional data thatis arranged on the first side of the two-dimensional surface to bedisplayed on the display screen.
 3. The image processing apparatus ofclaim 1, wherein the processing circuitry, in operation, positions thetwo-dimensional surface based on the position detected by the positiondetection circuitry and deforms the three-dimensional data such that thethree-dimensional data is arrangeable on the two-dimensional surface,arranges the deformed three-dimensional data on the first side of thetwo-dimensional surface, and applies rendering to, at least, therendering target data of the three-dimensional data that is arranged onthe first side of the two-dimensional surface to be displayed on thedisplay screen.
 4. The image processing apparatus of claim 3, whereinthe processing circuitry, in operation, cuts the three-dimensional datawith a surface parallel with the two-dimensional surface and arrangesthe cut surface on the two-dimensional surface.
 5. The image processingapparatus of claim 4, wherein if a plurality of positions detected bythe detection circuitry include a plurality of successive positions, theprocessing circuitry, in operation, generates the three-dimensional datafor each of the successive positions and combines the generatedthree-dimensional data at each of the successive positions so as togenerate three-dimensional data according to the successive positions.6. The image processing apparatus of claim 1, wherein the detectioncircuitry, in operation, detects a writing pressure applied to thepointer for each of a plurality of positions pointed to by the pointer,and the processing circuitry, in operation, sizes a three-dimensionalshape for each of the plurality of positions according to the writingpressure detected for each of the plurality of positions.
 7. The imageprocessing apparatus of claim 6, wherein the three-dimensional shape isselectable from among a plurality of different shapes.
 8. The imageprocessing apparatus of claim 1, wherein a position of thetwo-dimensional surface that is positioned by the processing circuitryis changeable.
 9. The image processing apparatus of claim 2, wherein thetwo-dimensional surface matches a sensor plane of the sensor.
 10. Theimage processing apparatus of claim 6, wherein the processing circuitry,in operation, selects a brush type, and changes a size of thethree-dimensional data in accordance with the brush type and the writingpressure.
 11. The image processing apparatus of claim 1, wherein theprocessing circuitry, in operation, selects a brush type, and athree-dimensional shape of the three-dimensional data varies dependingon the brush type.
 12. The image processing apparatus of claim 1,wherein the processing circuitry, in operation, applies surfacerendering to, at least, the rendering target data of thethree-dimensional data.
 13. The image processing apparatus of claim 1,wherein the processing circuitry, in operation, applies volume renderingto, at least, the rendering target data of the three-dimensional data.14. The image processing apparatus of claim 1, wherein the processingcircuitry, in operation, generates a two-dimensional image from adetection signal at the position detected by the detection circuitry.15. A non-transitory computer-readable medium storing a program that,when executed by a processor of an image processing apparatus, causesthe image processing apparatus to: generate three-dimensional data thatincludes a detected position on a sensor of the image processingapparatus that is pointed to by a pointer; position a two-dimensionalsurface relative to a display screen on which the three-dimensional datais displayable; and apply rendering to, at least, rendering target dataof the three-dimensional data that is arranged on a first side of thetwo-dimensional surface to be displayed on the display screen, therendering target data being defined based on the two-dimensionalsurface, the rendering not being applied to the three-dimensional datathat is arranged on a second side of the two-dimensional surface, thesecond side of the two-dimensional surface being opposite the first sideof the two-dimensional surface.
 16. The non-transitory computer-readablemedium of claim 15, wherein the program, when executed by the processorof the image processing apparatus, causes the image processing apparatusto: position the two-dimensional surface based on the detected positionand deform the three-dimensional data such that the three-dimensionaldata is arrangeable on the two-dimensional surface; arrange the deformedthree-dimensional data on the first side of the two-dimensional surface;and applies rendering to the, at least, rendering target data of thethree-dimensional data that is arranged on the first side of thetwo-dimensional surface to be displayed on the display screen.
 17. Thenon-transitory computer-readable medium of claim 15, wherein theprogram, when executed by the processor of the image processingapparatus, causes the image processing apparatus to: cut thethree-dimensional data with a surface parallel with the two-dimensionalsurface and arrange the cut surface on the two-dimensional surface. 18.The non-transitory computer-readable medium of claim 15, wherein theprogram, when executed by the processor of the image processingapparatus, causes the image processing apparatus to: if a plurality ofsuccessive positions is detected, generate the three-dimensional datafor each of the successive positions and combine the generatedthree-dimensional data at each of the successive positions so as togenerate three-dimensional data according to the successive positions.19. The non-transitory computer-readable medium of claim 15, wherein theprogram, when executed by the processor of the image processingapparatus, causes the image processing apparatus to: detect a writingpressure applied to the pointer for each of a plurality of positionspointed to by the pointer, and size a three-dimensional shape for eachof the plurality of positions according to the writing pressure detectedfor each of the plurality of positions.
 20. The non-transitorycomputer-readable medium of claim 19, wherein the program, when executedby the processor of the image processing apparatus, causes the imageprocessing apparatus to: select a brush type; and change a size of thethree-dimensional data in accordance with the brush type and the writingpressure.